Click any tag below to further narrow down your results
Links
This is NVIDIA's model card for Nemotron-3.5-Bolt-E4B-4GB, a small 4B-effective model post-trained from Gemma-4-E4B-it and aimed at fast, low-memory local voice and device assistants. The page is mostly deployment instructions, covering how to run it through llama.cpp, vLLM, SGLang, Ollama, LM Studio and several agent frameworks.
- Built for low memory and low time-to-first-token, with improved multi-turn tool use and instruction following over its Gemma base.
- Ships as a Q4_K_M quantized GGUF with up to 64K tokens of context, and supports toggling reasoning mode through the chat template.
- An optional DFlash drafter model enables speculative decoding for faster generation, and the model is licensed under OpenMDW-1.1 for commercial or non-commercial use.
@RatOrthodox on X
A developer built a free app that runs an AI model locally alongside about 1.5 million offline articles, including medical guides, FEMA material, survival manuals, and Wikipedia. The pitch is that it works entirely on your own computer with no internet connection needed.
- The app runs a full AI model on your own machine, so queries never leave the device.
- The bundled library holds roughly 1.5 million offline articles covering medical care, FEMA guidance, survival manuals, and Wikipedia.
- It is free, and the post frames it as a response to friends' growing interest in prepping.
The author uses a local LLM called pi that reads a global CAPABILITIES.md index in markdown, pointing to task-specific docs like SSHing into a studio Mac or driving a browser. Pi only loads detailed instructions when a task matches a capability, and new setups are documented and added to the index so agents grow more useful over time.
- A single symlinked AGENTS.md pointing to a CAPABILITIES.md index lets every project share one growing set of agent capabilities instead of duplicating setup per folder.
- Pi only loads detailed docs when a task matches one of the index's listed capabilities, keeping context small while still scaling to many tools.
- Adding a capability is just three steps (write the doc, add an index line, commit), so the library compounds over time as new setups get documented.
- The agent-docs repo is itself maintained almost entirely by AI, treating documentation upkeep as its own agent task.
This article walks through why and how to run large language models locally, covering privacy, cost, offline access, and control. It breaks down hardware needs, quantization, PC versus Mac setups, and starter software to get models up and running.
- Local open-source models (GLM-5.1, Kimi K2.6) now match or beat top cloud models like Claude Opus 4.6 on coding benchmarks, handling 70-80% of typical daily AI tasks.
- 4-bit quantization (Q4_K_M) shrinks memory needs to ~0.6-0.7 GB per billion parameters with no noticeable quality loss.
- RTX 4090 PCs outperform M3 Max Macs on speed (100-140 vs 40-60 tokens/sec) for sub-24GB models, but Mac Studios with unified memory (up to 512GB) can run massive 671B-parameter models entirely in RAM.
- A 3090 PC build costs $1,500-2,000 and draws 450W with heavy noise, versus a quieter Mac Studio pulling only ~60W while supporting far larger models.